Natural-language-instructed Industrial Task Execution

To effectively cooperate with a human, advanced manufacturing machines are expected to execute the industrial tasks according to human natural language (NL) instructions. However, NL instructions are not explicit enough to be understood and are not complete enough to be executed, leading to incorrected executions or even execution failure. To address these problems for better execution performance, we developed a Natural-Language-Instructed Task Execution (NL-Exe) method. In NL-Exe, semantic analysis is adopted to extract task-related knowledge, based on what human NL instructions are accurately understood. In addition, logic modeling is conducted to search the missing execution-related specifications, with which incomplete human instructions are repaired. By orally instructing a humanoid robot Baxter to perform industrial tasks “drill a hole” and “clean a spot”, we proved that NL-Exe could enable an advanced manufacturing machine to accurately understand human instructions, improving machine’s performance in industrial task execution. INTRODUCTION To effectively cooperate with a human, advanced manufacturing machines (machine for short) are expected to execute industrial tasks by following human instructions [1][2]. Typical communication methods for translating human instructions into a machine-readable commands, include visual-clue-based method which is involving in human body pose [3] , facial expression [4], and object [5], kinematics-clue-based method which is involving in action trajectory [6], action sequence [7] and movement dynamics [8], and bio-signal-clue-based method which is involving in electroencephalogram (EEG) pattern [9] and heart rate [10]. By using the above mentioned communication methods, a machine is enabled to understand human instructions and execute the human-assigned industrial task. However, all these methods were sharing the common disadvantages. First, their communication manners are unnatural. Only the expert users who have received prior trainings can use the above methods [3][7]. Second, the methods are ineffective in information transferring. Only some simple information patterns are defined, insufficiently describing task-related knowledge such as task procedures and human special requirements [6]. Third, these methods are procedure-complex and time-consuming. Their clue patterns need to be defined for encoding and decoding the task-related information embedded in human instructions [8]. Natural language (NL) communication is natural for that even the non-expert user without any prior training could interface with the machine [11][12]. NL is effective in information transferring for that NL includes sufficient standard linguistic patterns for delivering various information from a human to a machine [13][14]. NL communication is procedure-concise. Given NL is already an encoded information structure, the NL-based communication don’t need the encoding process, largely simplifying the information delivering procedures and reducing the information-transferring time [15][16]. Given the above advantages of NL communication, we adopted the NL communication method for human-machine cooperation in industrial task execution, aiming to improve the performance of an advanced manufacturing machine in following human instructions. With human NL instructions such as “slowly drill one hole in upper-right corner”, a machine is expected to understand the task goal such as “drill a hole” and also the execution-related specifications “location: upper right corner; requirement: slowly; action: drill”. However, some problems still exist, limiting a machine to directly understand a task from human NL instructions. First, NL instructions are various and ambiguous [17]. Determined by NL’s own characteristics such as polysemous, homophonic and divers, the expression manners of NL instructions are various. For example, “drill a hole” could also be expressed as “bore one hole”, “drilling one bore”, “create an unthreaded hole” and so on. Determined by instructors’ expression habits including referring, outlining and omitting, the NL instructions are usually ambiguous [18]. For example, in instruction “at the center point”, information such as “which